---
title: GraphOS reporting
subtitle: Send router operation metrics to GraphOS
description: Report GraphQL operation usage metrics from the Apollo GraphOS Router to GraphOS to enable schema checks and metrics visualization in GraphOS Studio.
redirectFrom:
  - /graphos/routing/graphos-reporting
---

The GraphOS Router and Apollo Router Core can report operation usage metrics to [GraphOS](/graphos/) that you can then visualize in GraphOS Studio. These metrics also enable powerful GraphOS features like [schema checks](/graphos/delivery/schema-checks/).

## Enabling usage reporting

You enable usage reporting in the router by setting the following environment variables:

```bash
export APOLLO_KEY=<YOUR_GRAPH_API_KEY>
export APOLLO_GRAPH_REF=<YOUR_GRAPH_ID>@<VARIANT_NAME>
```

<a id="usage-reporting-via-opentelemetry-protocol-otlp"></a>

### GraphOS tracing via OpenTelemetry Protocol (OTLP)

<MinVersionBadge version="Router v1.49.0" />

Prior to router v1.49.0, all GraphOS reporting was performed using a [private tracing format](/graphos/metrics/sending-operation-metrics#reporting-format) called Apollo Usage Reporting protocol.

As the ecosystem around OpenTelemetry (OTel) has rapidly expanded, Apollo evaluated migrating its internal tracing system to use an OTel-based protocol.

Starting in v1.49.0, the router can use OpenTelemetry Protocol (OTLP) to report traces to GraphOS. The benefits of reporting via OTLP include:

- A comprehensive way to visualize the router execution path in GraphOS Studio.
- Additional spans that were previously not included in Studio traces, such as query parsing, planning, execution, and more.
- Additional metadata such as subgraph fetch details, router idle / busy timing, and more.

Usage metrics are still using the Apollo Usage Reporting protocol.

<a id="configuring-usage-reporting-via-otlp"></a>
#### Configuring trace reporting via OTLP

You can enable trace reporting via OTLP by an option that can also configure the ratio of traces sent via OTLP and Apollo Usage Reporting protocol:

- In router v1.49-1.60, this is controlled using the `experimental_otlp_tracing_sampler` option and is disabled by default.

- In router v1.61, v2.x and later, this option is renamed to `otlp_tracing_sampler`.

- In router v2.x and later, this option is enabled by default.

The supported values of the OTLP sampler option are the following:

- `always_off`: send all traces via Apollo Usage Reporting protocol. Default for v1.x.
- `always_on`: send all traces via OTLP. Default for v2.x and later.
- `0.0 - 1.0`: the ratio of traces to send via OTLP (for example, 0.6 = 60% OTLP, 40% Apollo Usage Reporting protocol).

The OTLP sampler is applied _after_ the common tracing sampler. In the following example, the common sampler samples traces at 1% of all traffic. The OTLP sampler sets its ratio to 0.7. This results in 0.7% of all traffic having traces sent via OTLP, and the remaining 0.3% of all traffic having traces sent via Apollo Usage Reporting protocol:

```yaml title="router.yaml"
telemetry:
  apollo:
    # Send 0.7 OTLP / 0.3 Apollo
    otlp_tracing_sampler: 0.7

  exporters:
    tracing:
      common:
        # Sample traces at 1% of all traffic
        sampler: 0.01
```

<Note>

If your environment defines `OTEL_EXPORTER_OTLP_ENDPOINT` or `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`, they take precedence over your Router settings and may send traces to a different destination. Verify whether these variables are set and unset them if needed. For details, see the [OTLP exporter documentation](/router/configuration/telemetry/exporters/tracing/otlp).

</Note>

## Reporting field-level traces

In their responses to your router, your subgraphs can include [field-level traces](/federation/metrics) that indicate how long the subgraph took to resolve each field in an operation. By analyzing this data in GraphOS Studio, you can identify and optimize your slower fields:

<img
  src="../../../images/studio-trace.jpg"
  class="screenshot"
  alt="Viewing a trace in GraphOS Studio"
  width="500"
/>

Your subgraph libraries must support federated tracing (also known as FTV1 tracing) to provide this data.

- To confirm support, check the `FEDERATED TRACING` entry for your library on [this page](/federation/building-supergraphs/supported-subgraphs).
- Consult your library's documentation to learn how to enable federated tracing.
  - If you use Apollo Server with `@apollo/subgraph`, federated tracing support is enabled automatically.

### Subgraph trace sampling

By default, the router requests subgraph trace data from operations with a 1% sampling probability per operation. In most cases, this provides a sufficient sample size while minimizing latency for most operations (traces can affect latency because they increase the size of subgraph response payloads).

You can customize your router's trace sampling probability by setting the following options in your [YAML config file](/router/configuration/telemetry/overview/#yaml-config-file):

```yaml title="router.yaml"
telemetry:
  apollo:
    # In this example, the trace sampler is configured
    # with a 50% probability of sampling a request.
    # This value can't exceed the value of tracing.common.sampler.
    field_level_instrumentation_sampler: 0.5

  exporters:
    tracing:
      common:
        # FTV1 uses the same trace sampling as other tracing options,
        # so this value is also required.
        sampler: 0.5
```

<Note>

Because field-level instrumentation is dependent on general-purpose [OpenTelemetry tracing](/router/configuration/telemetry/exporters/tracing/overview), the value of `telemetry.apollo.field_level_instrumentation_sampler` cannot exceed the value of `telemetry.exporters.tracing.common.sampler`.

</Note>

### Disabling field-level traces

To completely disable requesting and reporting subgraph trace data, set `field_level_instrumentation_sampler` to `always_off`:

```yaml title="router.yaml"
telemetry:
  apollo:
    field_level_instrumentation_sampler: always_off
```

### Experimental local field metrics

Apollo Router can send field-level metrics to GraphOS without using FTV1 tracing. This feature is experimental and is not yet displayable in GraphOS Studio.
To enable this feature, set the `experimental_local_field_metrics` option to `true` in your router configuration:

```yaml title="router.yaml"
telemetry:
  apollo:
    experimental_local_field_metrics: true
```

## Advanced configuration

### `send_headers`

Provide this field to configure which request header names and values are included in trace data that's sent to GraphOS. By default, _no_ header information is sent to GraphOS as a security measure.

```yaml title="router.yaml"
telemetry:
  apollo:
    field_level_instrumentation_sampler: 0.01 # (default)
    #highlight-start
    send_headers:
      only: # Include only headers with these names
        - referer
    #highlight-end
```

**Supported values:**

<table class="field-table api-ref">
  <thead>
    <tr>
      <th>Value / Type</th>
      <th>Description</th>
    </tr>
  </thead>

<tbody>
<tr>
<td>

##### `none`

`string`

</td>
<td>

Set `send_headers` to the string value `none` to include _no_ header information in reported traces.

```yaml
send_headers: none
```

This is the default behavior.

</td>
</tr>

<tr>
<td>

##### `all`

`string`

</td>
<td>

Set `send_headers` to the string value `all` to include _all_ header information in reported traces.

```yaml
send_headers: all
```

**⚠️ Use with caution!** Headers might contain sensitive data (such as access tokens) that should _not_ be reported to GraphOS.

</td>
</tr>

<tr>
<td>

##### `only`

`array`

</td>
<td>

An array of names for the headers that the router _will_ report to GraphOS. All other headers are _not_ reported. See the example above.

</td>
</tr>

<tr>
<td>

##### `except`

`array`

</td>
<td>

An array of names for the headers that the router _will not_ report to GraphOS. All other headers _are_. Uses the same format as the `only` example above.

</td>
</tr>

</tbody>
</table>

### `send_variable_values`

Provide this field to configure which GraphQL variable values are included in trace data that's sent to GraphOS. By default, _no_ variable information is sent to GraphOS as a security measure.

```yaml title="router.yaml"
telemetry:
  apollo:
    field_level_instrumentation_sampler: 0.01 # (default)
    #highlight-start
    send_variable_values:
      except: # Send all variables EXCEPT ones with these names
        - first
    #highlight-end
```

**Supported values:**

<table class="field-table api-ref">
  <thead>
    <tr>
      <th>Value / Type</th>
      <th>Description</th>
    </tr>
  </thead>

<tbody>
<tr>
<td>

##### `none`

`string`

</td>
<td>

Set `send_variable_values` to the string value `none` to include _no_ variable information in reported traces.

```yaml
send_variable_values: none
```

This is the default behavior.

</td>
</tr>

<tr>
<td>

##### `all`

`string`

</td>
<td>

Set `send_variable_values` to the string value `all` to include _all_ variable information in reported traces.

```yaml
send_variable_values: all
```

**⚠️ Use with caution!** GraphQL variables might contain sensitive data that should _not_ be reported to GraphOS.

</td>
</tr>

<tr>
<td>

##### `only`

`array`

</td>
<td>

An array of names for the variables that the router _will_ report to GraphOS. All other variables are _not_ reported. Uses the same format as the `except` example above.

</td>
</tr>

<tr>
<td>

##### `except`

`array`

</td>
<td>

An array of names for the variables that the router _will not_ report to GraphOS. All other variables _are_ reported. See the example above.

</td>
</tr>

</tbody>
</table>

```yaml title="router.yaml"
telemetry:
  apollo:
    # The percentage of requests will include HTTP request and response headers in traces sent to GraphOS Studio.
    # This is expensive and should be left at a low value.
    # This cannot be higher than tracing->common->sampler
    field_level_instrumentation_sampler: 0.01 # (default)

    # Include HTTP request and response headers in traces sent to GraphOS Studio
    send_headers: # other possible values are all, only (with an array), except (with an array), none (by default)
      except: # Send all headers except referer
        - referer

    # Include variable values in Apollo in traces sent to GraphOS Studio
    send_variable_values: # other possible values are all, only (with an array), except (with an array), none (by default)
      except: # Send all variable values except for variable named first
        - first
  exporters:
    tracing:
      common:
        sampler: 0.5 # The percentage of requests that will generate traces (a rate or `always_on` or `always_off`)
```

### `errors`

You can configure whether the router reports GraphQL error information to GraphOS, and whether the details of those errors are redacted. You can customize this behavior globally and override that global behavior on a per-subgraph basis.

By default, your router _does_ report error information, and it _does_ redact the details of those errors.

- To prevent your router from reporting error information at all, you can set the `send` option to `false`.
- To include all error details in your router's reports to GraphOS, you can set the `redact` option to `false`.

Your subgraph libraries must support federated tracing (also known as FTV1 tracing) to provide errors to GraphOS. If you use Apollo Server with `@apollo/subgraph`, federated tracing support is enabled automatically.

To confirm support:

- Check the `FEDERATED TRACING` entry for your library on [the supported subgraphs page](/federation/building-supergraphs/supported-subgraphs).
- If federated tracing isn't enabled automatically for your library, consult its documentation to learn how to enable it.
- Note that federated tracing can also be sampled (see above) so error messages might not be available for all your operations if you have sampled to a lower level.

See the example below:

```yaml title="router.yaml"
telemetry:
  apollo:
    errors:
      subgraph:
        all:
          # By default, subgraphs should report errors to GraphOS
          send: true # (default: true)
          redact: false # (default: true)
        subgraphs:
          account: # Override the default behavior for the "account" subgraph
            send: false
```

#### Enabling extended error reporting

<div className="flex flex-row items-start gap-2 mt-2">
  <MinVersionBadge version="Router v2.1.2" />
  <PreviewFeatureBadge />
</div>

Enable richer error reporting via `preview_extended_error_metrics` and `redaction_policy` router configurations.

```yaml title="router.yaml"
telemetry:
  apollo:
    errors:
      preview_extended_error_metrics: enabled # (default: disabled)
      subgraph:
        all:
          # By default, subgraphs should report errors to GraphOS
          send: true # (default: true)
          redaction_policy: extended # (default: strict)
        subgraphs:
          account: # Override the default behavior for the "account" subgraph
            send: false
```

When enabled, the router sends metrics to Studio
with additional attributes including the `service` and `code` found in [GraphQL error extensions](https://spec.graphql.org/October2021/#sec-Errors.Error-result-format) (`errors[].extensions.service` and `errors[].extensions.code`, respectively).

Additional diagnostic capabilities available in Studio now support viewing errors by `service` or `code`. The `service` dimension refers to the subgraph or connector where the error originated from.
The `code` refers to the specific type of error that was raised by the router, federated subgraphs, or connectors.

#### Cardinality limitations

At scale, some metrics are known to hit cardinality limitations in the OTel reporting agent. When this happens, some of the metrics' attributes may no longer
be visible in Studio. To reduce cardinality warnings in your logs, adjust the Apollo metrics OTLP batch processor configuration to send reports more frequently. You can lower the `scheduled_delay` to reduce the cardinality stored in the router before the metric is flushed, as shown in the example:

```yaml title="router.yaml"
telemetry:
  apollo:
    metrics:
      otlp:
        batch_processor:
          scheduled_delay: 1s # default is 5s
```
